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ABSTRACT 

The misuse of outside consultants by most school 
districts has led to a disenchantment with inservice training in 
general. By using careful planning and evaluation techniques, the 
problems which have been caused by the inappropriate use of 
consultants and the unreal expectations of local school district 
personnel can be avoided. The process model described delineates the 
following steps needed to overcome this problem: specification of 
goals by local school districts, planning with school personnel 
regarding expected outcomes and work required, careful outlines of 
activities by the outside consultants, and well-planned evaluations 
of teacher and consultant behaviors. (Author) 
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Utilization of Consultants in Inservice Training 



In the book, Educational Evaluation and Decision Ha ving 
(Stuff lebean, et. al, 1971), the authors devote a section 
in the last chapter to a tonic they label, ’Therapy.” 

The paragraph that follows from that chanter was the gene- 
sis of this report. It states: 

. . . we need to devise a strategy to reach 
the consultants that are nov; working with 
practitioners on evaluation problems. These 
consultants, we have asserted, have been 
guilty of giving bad advice; indeed, they 
have frequently been unable to generate eval- 
uations that even minimally meet their own 
standards for inquiry. Since these consult- 
ants are in the main university professors, 
some pattern of national institutes or semi- 
nars in which these problems can be discus- 
sed and some training in new approaches 
offered seems to be in order, (p. 345) 

The Laboratory of Educational Research, University of the 
Pscific has frequently been asked to perform evaluations, 
many of which unfortunately, were "patch-ur>" jobs of evalu- 
ations begun by others who were not competent to complete 
them. The problem became very apparent, however, when the 
Laboratory received a grant from the State in 1970 to write 
some dissemination documents based on evaluative reports, 
frequently written by college or university orofessors. 



After working with the state officials for a very short time, 
it was apparent that the great majority of the orofessors 
who had taken on the evaluative role in the various projects 
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within the Bureau's jurisdiction were (a) either unable to 
carry out a proper evaluation for lack of skills, or (b) had 
taken a task which was ."i-nadecjuately funded and were doing 
only enough to justify the money which they were being paid, 
or (c) found the guidelines published by the State to be too 
general to be helpful , It also became apparent that many 
of the local school evaluators were doing a minimal job of 
evaluating because (a) they were over-loaded with work, 

(b) they were unable to interpret the directives ccnincr 
from the State, or (c) they lacked the skills necessary to 
perform an evaluation. 

In any case the project evaluations were being sorely ne- 
glected or poorly done. In addition to the work with this 
State Bureau, on two occasions the Laboratory was called 
in to complete Title III evaluations which had been begun 
by a person or group of persons who passed themselves off 
as evaluators but who wore unable to perform the tasks 
properly. The :, patch-up ,s type of evaluation which the 
Laboratory was able to perform was not ud to the standards 
which were generally adhered to ; although the local dis- 
tricts were generally quite grateful to simply have some- 
one who could help them conolete the evaluation requirements 
of their grants. 
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Faced with this situation, the need for a strategy to over- 
come this problem was evident. Since the tine frame in which 
the Laboratory was working was a tight one, it was imperative 
that whatever training that could be provided be done as ex- 
peditiously as possible, and that only a minimal amount of 
skill could be expected from many of the evaluators. 

Betz (1969), Tuttle and Ciccone (1969) and the American As- 
sociation of Junior Colleges (1967) have indicated that the 
role of a consultant requires more than simply appearing for 
a day and talking from the "ton of the head" or accepting 
data unquestioninglv and submitting it to a canned computer 
program, interpreting the data in a mechanical fashion and 
vxriting a report. It was apparent from the reports made 
available by the State that these v?ere the main uses being 
made of consultants by the local districts , however . 

Since the Laboratory was faced with deadlines and little or 
no information upon which to base the dissemination reports, 
a dual problem faced the staff. One was to obtain enough 
data from the previous projects to enable them to write the 
reports for the 1969-70 year, and the second, to ensure that 
the reports based on the data collected during 1970-71 v/ould 
be collected, analyzed and reported properly. 
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The strategy devised to accomplish these goals was two-fold: 
first, as consultants for the State Eureau, the staff arranged 
to make on-site visits to each project, discus? the needs of 
the Bureau for documented information in written form; and 
second, as consultants for the local districts, to help pre- 
pare an outline and give advice which would enable them to 
provide all the necessary information to complete the dis- 
semination reports for 1970-71 in their written documents. 

As simple as it sounds, the outline was the conventional one 
shown here; 




Figure 1. Sequence of events in preparing evaluation reports. 

It was surprising to find some project directors did not 
even know what their objectives wo re and others had seen lit- 
tle reason to keep documented records of their program activ- 
ities since 'they knew what they were doing." Many hau not 
thought to check for congruence between objectives, program 
activities and evaluative criteria, and several had written 
many objectives for which there was no program element which 
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would insure its being reached nor a criterion against 
which to measure it. One recommendation which grew out of 
these findings was that Stake's model (1967), which empha- 
sises congruence,- should be included in any inscrvice train- 
ing given to evaluators. 

The criteria by which the Laboratory was to be judged for 
the work of preparing the 1969-70 reports was the degree 
to which the essential information needed to write the dis- 
semination documents could be elicited post hoc and reduced 
to written form. Although not happy with the minimal amount 
of data collected from some projects, and the questionable 
validity of some of that which was received, the Laboratory 
staff was able to complete the project and the 1969-70 re- 
ports were accepted for printing and dissemination. 

The second, and most important part of the strategy for in- 
service training was to insure that the costly on-site visits 
to collect post hoc information of questionable validity 
would be avoided in 1970-71. To accomplish this task, the 
following strategy was adopted, using the techniaues and 
ideas which seemed most appropriate in the papers cited 

earlier and others (Stuff lebeam, 1966; Provus, 1969). 
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•Oiiis strategy required on the p*wt of both tho consultant 
and. the person requesting consultative hcln a co:nmitnont 
to prepare before the actual .inservice training (since we 
consider each consultation as an insorvice training activity) 
a recommendation made in the American Junior Colleges paner. 
Also the strategy requires that a clear explanation of what 
is to be expected of each party be made prior to the train- 
ing session. In many cases during 19G9-70 consultants had 
been called to particular jobs by project directors and ar- 
rived on tho day that they were to perform their activities 
with a general outline of what they were to present but 
with no preparation required of the teachers who were to 
be involved. In these cases it is not at all surnrising 
that the results were less than expected. 

One other factor that all consultants have not considered 
in the past is the fact that every consulting job should 
involve both preparation prior to the consultation and a 
follow up in a form of written reports of what was accom- 
plished and/or additional training sessions. This follow- 
up activity element is an essential oart of the training 
strategy devised by the Laboratory, and "one-shot" train- 
ing sessions are not accepted since they violate goo J 
learning theory and practice. 
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The preparation that was done by the Laboratory was the 
writing and printing of materials to be sent to each pro- 
ject director/evaluator indicating the hinds of evaluation 
that would be considered minimal. Tl.c concent of congru- 
ence was emphasized in terms of examining the objectives, 
program activities, criteria for evaluating the success or 
failure of the project and the actual outcomes of the 
project itself. To be certain these elamentr were related 
lcyicallv and that the objectives were not only measure- 
able but being measured by instruments which actually 
measured what was being proposed in the objective, several 
examples of good and bad objectives and criteria were pre- 
sented. In many cases it was found that objectives had. 
been written which did not in any way correspond to the 
program and that the evaluation used v/as totally invalid. 
Project directors and evaluators were shewn the discrepan- 
cies during on-site visits. 'lest of them indicated that 
they thought that they had to use the state-mandated tests 
even though they realized that they were frequently not 
really measuring the objectives of the program. .Many wo re 
excited and relieved to know that the choice of the criterion 
was one in which they had a choice. Whan it was pointed 
out that the success or the failure of the project was 
going to hinge on the evaluation based on these instru- 
ments, the necessity for careful selection of criteria was 
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very obvious to them. This initial preparation on the 
part of the Laboratory, then was preparation of a state- 
ment indicating the need for clearly stated, and con- 
gruent objectives, program activities , criteria and out- 
comes. 

Following the preparation and dissemination of this state- 
ment the Laboratory prepared a list of the needs and con- 
straints which .it faced. The Bureau also prepared a list 
of its needs and constraints, and the local px-oject evalu- 
ators were asked to prepare a list of their needs and con- 
straints. In this case the Bureau, working with the Labora- 
tory, prepared a new application blank in which the objectives, 
program description and evaluation plans were vary carefully 
delineated, and each element related to the budget. 

Each one of the three groups of participants were then asked 
to come to one or two ir.sorvico meetings lasting for a full 
day. At this meeting the Bureau and the Laboratory stated 
as clearly and precisely as possible the reason for and the 
use to be made cf each of the particular kinds of informa- 
tion requested in tho evaluation forms. Local oroject 
directors stated many valid reasons for not being able to 
comply with some requests made by the State and/or labora- 
tory. The local project directors and evaluators stated 
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on several occasions that they had had no idea of whv 
the Bureau or Laboratory needed certain kinds of infor- 
mation.- and that they had considered the collecting of 
much of the information ar ' •■v.ay work' - ' unf-.l thov 
saw the reason for it. Once they understood the need, 
they all indicated their willingness to cooperate in pro- 
viding the essential information. 

turn, both the Bureau and Laboratory rc— examined their 
needs and constraints to utilize existing data collection 
procedures and data being generated in the local districts 
wherever possible to minimize the amount of paper work 
and man hours needed to produce the information needed by 
these agencies. After three days of hard work, a form 

collecting all the essential data needed k>v the Bureau 
and which would require a minimal amount of additional 
effort on the part of the local "'reject directors was de- 
vised. The form also maee the work cf the Bureau staff, 
which was undermanned, much easier by reouirina much 
snorter and more easily read interim reports. This savings 
1*' time was accompli shccl by using a modification of the 
discrepancy model for process evaluation (Provus, 1969> . 
This allowed for a very quick examination by the Bureau 
staff of each of the projects ir. each quarterly report 
and allowed them to determine which programs were in 
difficulty and which were progressing on schedule so that 

■*4 
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thoy could concentrate their efforts in those areas where 
they were most needed. !'7hon evaluation problems were 
noted by the Bureau staff, they were referred to the Lab- 
oratory staff who provided the project evaluator with help 
on the problems which he encountered. 

It should be mentioned that in many cases the university 
professors have been very poorly trained in terms of evalu- 
ation processes as Stufflebeam and others (1971) have indi- 
cated. Many of them have had little or no statistical train- 
ing nor any formal training in the evaluative procedures 
which have developed during the past decade. These people 
needed consultive help in planning appropriate evaluation 
for their particular projects. Many did not understand 
the concept of process evaluation since this concent had 
developed subsequent to the time they had completed their 
education. Therefore, at those meetings, the concepts of 
a timeline and a process evaluation involving quarterly 
reporting was explained and actually begun. 

One of the problems that evaluators have failed to deal 
wit); adequately and which is described in Educational Eval- 
uation and Decision Making is public relations. The stra- 
tegy used to attack this problem by the Laboratory '- T as to 
actively involve all participants in the planning of the 
evaluative design, and to emphasize again and again that 
* 
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the purpose of evaluation was not to prove or disprove, 
but to improve the respective projects. This approach 
proved invaluable in getting the cooperation of the par- 
ticipants by changing their perspective from a defensive 
one to a cooperative one. At these inservice meetings the 
kinds of product evaluation which would be needed from 
each cf the projects was also established. Additional tine 
v/as provided whore necessary fer local project directors/ 
evaluators to consult with the Laboratory staff for product 
evaluation. The follow-up consultations and written doc- 
umentation of them T *cre produced. 

The criteria for evaluating the success of the Laboratory 
in reaching the second goal was in terns of the number of 
projects which were able to complete their evaluations on 
schedule and in sufficient detail that dissemination doc- 
uments could be prepared without having to make on-site 
visits to obtain additional information. Table I shov’s 
the comparison between the first and second years in terms 
of the percentage of projects which met this criterion. 
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TA3LE I, 

Number of Projects which Provided Adequate Information 
to Prepare Dissemination Documents in 1569-70 and 1970-71 



Year 


No. cf 


2*o . with 


Per Cent 




Projects 


Adequate 


of 






Evaluation 


Adequate 

Evaluation 


1969-70 


18 


5 


23.3% 


1970-71 


11 


o 


81.8% 



During the first year, only tvro projects had been writ- 
ten well enoucrh that reports could be prepared 'without hav- 
ing to go to the project site and obtain additional infor- 
mation. During the second year, 1970-71, only two projects 
had to be contacted in order to gather additional informa- 
tion. It should be noted that in these two cases, rather 
than starting from zero information as we had in 1969-70 
the problems were in the nature of interpretation of data 
and could be handled by telephone. In one instance the 
experimental design used was a three way analyses of vari- 
ance with three or four levels for each variable and with 
a sample size of only 72, which resulted in many empty 
cells. When this fact was called to the attention of the 
project evaluator, he reanalyzed the data appropriately 
and provided useful information. In the second case. 
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the professor involved relied on the computer program 
without carefully checking the printout. Our analysis 
indicated that there was an obvious error in their computer 
program. The professor, not being sophisticated in statis- 
tical techniques, had not caught the error, and both ho and 
his computer people were chagrined to find that their pro- 
gram was in error. The data was reanalyzed on another com- 
puter and useful information was obtained. 

The figures in Table I, therefore, do not reflect adequately 
the qualitative difference in the data collected for the 
tv/o years. The strategy which employed had been remark- 
ably successful in upgrading the evaluations provided to 
the State. The work, however, was a good beginning but 
one which requires more refinement. 

In summary, the problem is largely one in which trained 
researchers or evaluators make assumptions that people in 
the field have the same kind of expertise and the same 
desire for evaluative data that they do. This is not the 
case. Project directors are interested in implementing an 
idea. They are much less interested, and frequently 
threatened, when someone mentions evaluation. They fre- 
quently are convinced that evaluation equals spying, and 
see no need for an evaluation component in their project. 

J5 
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The poor job done by inadequately trained evaluators, many 
of whom are hichly skilled researchers, hnr. created a bad 
image for us. As an example the training of auditors for 
the Title VII evaluations has not beer, nearly as produc- 
tive as it should. The training consists of two and one 
half days of concentrated study, assuming a very lov; base 
of knowledge of the participants. No follow-uo training 
is provided and many of those certified are overwhelmed 
with the concentratad training given v/ith no time for 
thinking through the process and having an opportunity to 
raise questions. As Dr. Kenneth Hopkins of the Labora- 
tory of Educational Research, University of Colorado, said 
to me during the first training session in Washington, 

D.C. ’’The concept is good, but the tragedy is that the 
participants' backgrounds arc such that they need such 
simple, basic training.' An auditor with that minimal 
amount of skill will hardly be able to adequately audit a 
complex project. 

To properly use consultants in evaluation then reguires : 

1. a careful preparation of the consultant in terms 
of evaluation strategies as well as research 
strategies 

2, a careful preparation of the consultant in 
human relations skills to overcome the bad 
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image evaluation has engendered in the oast 
because of poorly trained and inadequately 
financed evaluators . 

3. a careful preparation of the evaluator to 
insure that ho knows when adequate resources 
are provided to do a good job of evaluation 
and the guts to turn down those evaluations 
which do not provide these resources 

4. a recognition that re-training and/or train- 
ing of most people who will be doing evaluations 
must ia done over a longer period of time rather 
than the 'one-shot 1 '' two to five day intensive 
training programs nov; in vogue. Learning a new 
skill, to be really assimilated thoroughly, 
requires some time for practice and incubation. 
Follow-up activities must be built into any 
program for training or using consultants , and 
the time frame should be over a period of weeks 
and months rather than days. 
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